@philo-he, @zhouyuan could you have a look at the PR?

@sezruby, thanks for the PR. This fix makes sense. I recall there's a related issue that occurs at compile time when an external project introduces the Gluten JAR as a dependency: a Scala type mismatch caused by the Maven Shade Plugin not rewriting ScalaSignature annotations. My understanding is this PR also fixes that case (see https://chungmin.hashnode.dev/unraveling-a-scala-type-mismatch-mystery).

One small concern is potential Arrow version conflicts, since these packages are no longer shaded. That said, the memory and vector APIs should be stable across minor versions, so I assume the risk should be low in practice.

cc @zhztheplayer

That said, the memory and vector APIs should be stable across minor versions

This sounds a real risk. Moving forward, can we completely remove Arrow from the bundled Gluten Jar, and let user rely on Spark's bundled Arrow instead?

I assume we don't have any customized Arrow code now: #12130

can we completely remove Arrow from the bundled Gluten Jar, and let user rely on Spark's bundled Arrow instead? I assume we don't have any customized Arrow code now: #12130

Worth doing, but a couple of things worth confirming first:

#12130 removed the dead Arrow-CSV / Arrow-Dataset JVM paths. That's progress, but there are still ~34 files under gluten-arrow/ and elsewhere that call org.apache.arrow.* (ColumnarBatches, ArrowColumnVector, ArrowWritableColumnVector, ColumnarBatchSerializer, JNI columnar batch bridges, SparkArrowUtil, etc.). Those code paths are alive on the Velox hot path and the bundled Arrow is what they currently resolve against.

If we drop the bundled Arrow:

1. arrow.version in gluten's poms would need to align with whatever the lowest-common-denominator target Spark ships — Arrow 12 for Spark 3.5 (DBR 16.4 ships 12.0.1, vanilla Spark 3.5.x ships 15.0.0), Arrow 18 for Spark 4.x. Compile profile per Spark version, similar to how <spark.version> is already keyed.
2. No 15.0.0-gluten source patches relied on — needs confirmation. The arrow-gluten.version = 15.0.0-gluten in pom.xml suggests there's at least some customization, even if the dead code in [MINOR][VL] Remove dead Arrow-CSV / Arrow-Dataset JVM code paths #12130 was the main consumer. Worth a sweep for any remaining call site that depends on a non-public Arrow API.
3. Testing matrix — minor: confirm no NoSuchMethodError on the Spark distros gluten claims support for, especially DBR / Cloudera flavors that ship older Arrow.

So I think it's the right long-term direction. As an immediate fix this PR makes the current shading approach internally consistent (which is independently a valid bug fix, since the partial-shading is a latent bug regardless of whether Arrow gets unbundled later). Happy to take either path — let me know if you'd prefer I close this and pursue the Arrow-unbundling work instead, or merge this as the short-term fix and treat unbundling as a follow-up.

a Scala type mismatch caused by the Maven Shade Plugin not rewriting ScalaSignature annotations

@philo-he Good link, the partial-shading also breaks downstream Scala consumers that pull the gluten jar as a dep, since their compile-time Arrow type doesn't match what gluten expects across the API boundary. This PR fixes that as a side effect by keeping the boundary types unshaded so they match the public Apache Arrow types every other Scala consumer compiles against. cc author @clee704

arrow.version in gluten's poms would need to align with whatever the lowest-common-denominator target Spark ships

Exactly. This is the right approach. Or we just adjust the version accroding to spark.version to match Spark's Arrow.

No 15.0.0-gluten source patches relied on — needs confirmation.

Let's track back on the history to confirm that. cc @jinchengchenghh

@sezruby thanks for the fix! Could you please also share the arrow usage used in your case - does those libs also shade their arrow API?

This affects any Spark workload that combines gluten with another library using Arrow C-Data

Could you please also share the arrow usage used in your case - does those libs also shade their arrow API?

Sure — the trigger is Lance Java (org.lance.spark.write.LanceDataWriter), which uses the standard Arrow C-Data interface to ship Spark RecordBatches into a native (Rust) Lance writer. The relevant call is:

try (ArrowArrayStream stream = ArrowArrayStream.allocateNew(allocator)) {
  Data.exportArrayStream(allocator, sparkArrowReader, stream);
  Fragment.create(datasetUri, stream, params);
}

allocator here is org.apache.arrow.memory.BufferAllocator (the public Arrow type). Lance Java does not shade Arrow — it depends on org.apache.arrow.* directly. Arrow Java 15.0.2 is pulled in transitively via lance-core.

Three observations from running this on a Spark cluster that ships the gluten-velox bundle on the AppClassLoader:

1. JVM resolves org.apache.arrow.c.ArrowArrayStream to gluten's bundled copy (gluten on AppClassLoader, our jar on the child MutableURLClassLoader).
2. Gluten's ArrowArrayStream.allocateNew signature is allocateNew(org.apache.gluten.shaded.org.apache.arrow.memory.BufferAllocator).
3. Lance's call passes org.apache.arrow.memory.BufferAllocator → NoSuchMethodError.

Other public-API Arrow C-Data callers on the same classpath would hit the same shape — Iceberg's Arrow vector layer, Snowflake JDBC's Arrow result decoder, etc. — but I haven't tested those directly, just calling out that the issue isn't Lance-specific.

Workarounds tried at the user-jar layer before this PR: userClassPathFirst=true (cascading LinkageError on Arrow Field because Spark internals on AppClassLoader pre-load Arrow before the user-jar takes over); shade-relocating org.apache.arrow.c → org.lance.shaded.arrow.c in the user fat jar (gets past NoSuchMethodError but trips UnsatisfiedLinkError because libarrow_cdata_jni.so has the JNI natives bound to org/apache/arrow/c/jni/JniWrapper). Neither is a clean user-side fix, which is why this needs to be addressed at gluten's shading layer (or via the unbundling path you proposed above).